Active Database Learning

Learning from Past Query Processing— In today’s databases, the answers to past queries barely benefit processing future queries. The query answers and the work performed for processing queries—such as I/O and computations—are no longer used after returning query answers. Database Learning (DBL) [4] proposes to change this paradigm in an approximate query processing (AQP) context. DBL uses its knowledge acquired from past queries and their answers for speeding up future query processing. The more queries are processed, the smarter a system becomes; thus, the faster it can process new queries. DBL is based on the observation that the answers to queries (even if they access different subsets of rows and columns) stem from the same underlying distribution (which produced the entire data). Therefore, an answer to each query reveals a piece of information about this underlying distribution, which can be used to construct a concise statistical model of the distribution. Using a statistical model could bring a significant performance benefits. In an ideal case where the model precisely captures the underlying distribution, we could answer queries by analytically evaluating the model, instead of reading and processing terabytes of raw data. Even an imprecise model can still be beneficial; for instance, one can use a small sample of the entire data to quickly produce a sample-based answer [1], which can then be combined with the model to produce a more accurate approximate answer. Our prototype [4]—that uses the maximum entropy principle for model construction and SparkSQL for distributed computations—showed 73.7% support of real-world query workloads and 10.43 times of possible speedup compared to existing AQP systems (at the same accuracy level).